NUCLEAR NORMS OF RANK 2 MATRICES FOR SPECTRAL CONDITION 
NUMBERS OF FULL RANK LINEAR LEAST SQUARES SOLUTIONS 

JOSEPH F. GRCAR* 



Abstract. The condition number of solutions to full rank linear least-squares problem are shown to be given 
by an optimization problem that involves nuclear norms of rank 2 matrices. The condition number is with respect 
to the least-squares coefficient matrix and 2-norms. It depends on three quantities each of which can contribute ill- 
conditioning. The literature presents several estimates for this condition number with varying results; even standard 
reference texts contain serious overestimates. The use of the nuclear norm affords a single derivation of the best 
known lower and upper bounds on the condition number and shows why there is unlikely to be a closed formula. 
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1. Introduction. 



1.1. Purpose. Linear least-squares problems, in the form of statistical regression anal- 
yses, are a basic tool of investigation in both the physical and the social sciences, and conse- 
quently they are an important computation. 

This paper develops a single methodology that determines tight lower and upper esti- 
mates of condition numbers for several problems involving linear least-squares. The con- 
dition numbers are with respect to the matrices in the problems and scaled 2-norms. The 
problems are: orthogonal projections and least-squares residuals (Grcar] |2010b| l, minimum 
2-norm solutions of underdetermined equations ( |Grcar| |2010a| l, and in the present case, the 
solution of overdetermined equations 



arg min \\b — Au\\2 



A t Ax = A t b, 



(1.1) 



where A is an m x n matrix of full column rank (hence, m > n). Some presentations of error 
bounds contain formulas that can severely overestimate the condition number, including the 
SIAM documentation for the LAPACK software. 

This introduction provides some background material. Section[2]discusses definitions of 
condition numbers. Section [3] describes the estimate and provides an example; this material 
is appropriate for presentation in class. Section |4] proves that the condition number varies 
from the estimate within a factor of v2- The derivation relies on a formula for the nuclear 
norm of a matrix. (This norm is the sum of the singular values including multiplicities, and 
is also known as the trace norm.) Section[5]examines overestimates in the literature. Section 
[^evaluates the nuclear norm of rank 2 matrices (lemma |6~3j ). 



1.2. Prior Work. Ever since Legendre (1805) and |Gauss (1809) invented the method 
of least-squares, the problems had been solved by applying various forms of elimination 
to the normal equations, A 1 Ax = A l b in equation ( |l.l[ ). Instead, Golub ( 1965 i suggested 
applying Householder transformations directly to A, which removed the need to calculate 
A* A. However, Golub and Wilkinson (1966 p. 144) reported that A 1 A was still "relevant 
to some extent" to the accuracy of the calculation because they found that A 1 A appears in 
a bound on perturbations to x that are caused by perturbations to A. Their discovery was 
"something of a shock" ( |van der Sluis||1973] p. 241). 
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The original error bound of |Golub and Wilkins on] ( ] 19661 p. 144 , eqn. 43) was difficult 
to interpret because of an assumed scaling for the problem. [Bjorck ( 1967 pp. 15, 17, top) 



derived a bound by the augmented matrix approach that was suggested to him by Golub. 



Wedin ( 1973| pp. 224-226) re-derived the bound from his study of the matrix pseudoinverse 
and exhibited a perturbation to the matrix that attains the leading term. Van der Sluis (1975, 
p. 251, eqn. 5.8) also derived Bjorck's bound and introduced a simplification of the formula 
and a geometric interpretation of the leading term. Bjor ck] ( ["1996| p. 31, eqn. 1.4.28) later 



followed Wedin in basing the derivation of his bound on the pseudoinverse. Malyshev (2003 
p. 1 189, eqn. 2.4 and line -6) derived a lower bound for the condition number thereby proving 
that his formula and the coefficient in Bjorck's bound are quantifiably tight estimates of the 
spectral condition number. In contrast, condition numbers with respect to Frobenius norms 
have exact formulas that have been given in various forms by |Geurts] ( |1982| l, |Gratton| ([T996 ), 



and Malyshev (2003). 



2. Condition numbers. 

2.1. Error bounds and definitions of condition numbers. The oldest way to derive 
perturbation bounds is by differential calculus. If y = f(x) is a vector valued function of 
the vector x whose partial derivatives are continuous, then the partial derivatives give the best 
estimate of the change to y for a given change to x 

Ay = f{x + Ax)-f(x)^J f (x)Ax (2.1) 

where Jf(x) is the Jacobian matrix of the partial derivatives of y with respect to x. The 



magnitude of the error in the first order approximation (2.1 1 is bounded by Landau's little 
o(|| Ax||) for all sufficiently small || Ax HQ Thus Jf(x) Ax is the unique linear approximation 
to Ay in the vicinity of x r\ Taking norms produces a perturbation bound, 

\\Ay\\ < \\J f (x)\\ ||Ax||+o(||Aa:||). (2.2) 



Equation (2.2 1 is the smallest possible bound on ||Ay|| in terms of ||Aa;|| provided the norm 



for the Jacobian matrix is induced from the norms for Ay and Ax. In this case for each x 



there is some Ax, which is nonzero but may be chosen arbitrarily small, so the bound (2.2i 



is attained to within the higher order term, o(||Ax||). There may be many ways to define 



condition numbers, but because equation (2.2i is the smallest possible bound, any definition 



of a condition number for use in bounds equivalent to ( |2.2| i must arrive at the same value, 
Xy( x ) = 11^/(^)111^] The matrix norm may be too complicated to have an explicit formula, 
but tight estimates can be derived as in this paper. 

2.2. One or separate condition numbers. Many problems depend on two parameters 
u, v which may consist of the entries of a matrix and a vector (for example). In principle it 
is possible to treat the parameters altogether]^] A condition number for y with respect to joint 



1 The continuity of the partial derivatives establishes the existence of the Frechet derivative and its representation 
by the Jacobian matrix. The definition of the Frechet derivative is responsible for the error in equation (2.1) being 
o(||Aa;||). The order of the error terms is independent of the norm because all norms for finite dimensional spaces 
are equivalent {Stewart and Sun||l990| p. 54, thm. 1.7). 

2 Any other linear function added to Jf(x) Ax differs from Ay by 0(\\ Ax||) and therefore does not provide a 
o(||Ax||) approximation. 

3 A theory of condition numbers in terms of Jacobian matrices was developed b y Rice 1 196 6| p. 292, t hm. 4). 
Recent presentations of the formula X y( x ) = \\Jf( x )\ \ 316 given by |Chaitin-Chatelin and Frayssej |l 996| p. 44), 
[Deuflhard and Hohmann||2(j03] p . 27), Quartero TrIet^|2CJ00] p. 39), and |Trefethen and Bau|{l$?7| p^0r 

4 As will be seen in tab"Ie p.l| Gratton 1 1996 1 derived a joint condition number of the least-squares solution with 
respect to a Frobenius norm of the matrix and vector that define the problem. 
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changes in u and v requires a common norm for perturbations to both. Such a norm is 

max{||Au||, ||Au||} . (2.3) 
A single condition number then follows that appears in an optimal error bound, 

||Ay|| < ||J/(tt,u)|| max{||Au||, ||Au||} + o (max {||Au||, ||A«||}) . (2.4) 

The value of the condition number is again Xy( u > v) — \\Jf(u, v)\\ where the matrix norm is 
induced from the norm for Ay and the norm in equation ( |2.3| l. 

Because u and v may enter into the problem in much different ways, it is customary to 
treat each separately. This approach recognizes that the Jacobian matrix is a block matrix 

J f (u,v) = [ J h {u) J h (v) ] 

where the functions fi(u) = f(u,v) and /2(f) = f(u,v) have v and u fixed, respectively. 
The first order differential approximation \2.\\ is unchanged but is rewritten with separate 
terms for u and v, 

Ay w J h (u) Au + J h (v) Av . (2.5) 
Bound ( |2.4[ ) then can be weakened by applying norm inequalities, 

\\Ay\\ < \\J fl {u)Au + J h (v)Av\\ + o (max { || Au\\, ||Au||}) 

< (II^WII + \\Jh(v)\\) max{\\Au\\, \\Av\\} (2.6) 

+ o (max{||Au||, ||Au||}) . 

The coefficients Xy( u ) — an d Xy{ v ) — ll^/ 2 ( v )ll are me separate condition num- 

bers of y with respect to u and v, respectively. 

These two different approaches lead to error bounds ( |2.4| |2.6[ ) that differ by at most a 
factor of 2 because it can be shown (Grcar, 2010b) 

Xy{u)+ 2 X y {v) < Xy (u, v) < Xy (u) + Xy (v) . (2.7) 

Thus, for the purpose of deriving tight estimates of joint condition numbers, it suffices to 
consider Xy( u ) an d Xy( v ) separately. 

3. Conditioning of the least-squares solution. 

3.1. Reason for considering matrices of full column rank. The linear least-squares 
problem (JTTTJ does not have an unique solution when A does not have full column rank. A 
specific x can be chosen such as the one of minimum norm. However, small changes to A 
can still produce large changes to xj^jln other words, a condition number of x with respect to 
rank deficient A does not exist or is "infinite." Perturbation bounds in the rank deficient case 
can be found by restricting changes to the matrix, for which see Bjorck] ( |1996| p. 30, eqn. 



1.4.26) and Stewart and Sun ( 1990 p. 157, eqn. 5.3). That theory is beyond the scope of the 



present discussion. 



If A does not have full column rank, then for every nonzero vector z in the right null space of the matrix, 
(A + bz t )(z/z t z) = b. Thus, a change to the matrix of norm || b|| 2 H2II2 changes the solution from A^bto 2/||z|||. 
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3.2. The condition numbers. This section summarizes the results and presents an ex- 
ample. Proofs are in section [4] It is assumed that A has full column rank and the solution 
x of the least-squares problem (JTTTJ is not zero. The solution is proved to have a condition 
number \x (A) with respect to A within the limits, 



k 2 J[vta,n(9)} 2 + 1 



< Xx{A) < K 2 [vtan(0) + 1] , 



(3.1) 



where k 2 , V, and 9 are defined below; they are bold to emphasize they are the values in the 
tight estimates of the condition number. There is also condition number with respect to b, 



Xx(b) = Vsec(0 



(3.2) 



These condition numbers Xx(A) and Xx(b) are for measuring the perturbations to A, b, and 
x by the following scaled 2-norms, 



|A6|| a 



\Ax\\ 2 

\\*h 



Like equation (2.6 1, the two condition numbers appear in error bounds of the formj^] 



Ax\\ 2 „ /A JAA\\ 2 , „.JA6|| 2 , / /||A^|| 2 ||A6|| 2 

: ho max 



<Xx{AY 



Xx{bV- 



Flh - ' ' ||A|| a — ' ||b|| 2 V 
where x + Ax is the solution of the perturbed problem, 



Uh ' \\b\\ 



Ax = argmin||(6 + Ab) - (A + AA)u\\ 2 



The quantities n 2 , V, and 9 in the formulas (3.1 3.2 1 are 



\Ax\ 



IF 2 ^mir 



tan(0) 



\\rh 
\Ax\\ 2 



(3.3) 



(3.4) 



(3.5) 



(3.6) 



where k 2 is the spectral matrix condition number of A (cr m ; n is the smallest singular value of 
A), V is van der Sluis's ratio between 1 and k 2 Q^ i s tne an gle between b and col(^4)J^]and 
r = b — Ax is the least-squares residual. 

1 . k 2 depends only on the extreme singular values of A. 

2. 9 depends only on the "angle of attack" of b with respect to col (A). 

3. If A is fixed, then V depends on the orientation of b to col (A) but not on 9^ 



Please refer to Figure 3.1 as needed. If col(^4) is fixed, then k 2 and V depend only on the 
singular values of A, and 9 depends only on the orientation of b. Thus, k 2 and 9 are separate 
sources of ill-conditioning for the solution. If Ax has comparatively large components in 
singular vectors corresponding to the largest singular values of A, t hen V 
condit ion number Xx(A) depends on n 2 which was the di scovery of 
( 19661. Otherwise, k 2 "plays no role" (van der Sluis 1975 p. 251). 



k 2 and th e 



Golub and Wilkinson 



6 The constant denominators ||A||2 and 
magnitude of the ter ms is pertinent. 



j2 could be discarded from the o terms because only the order of 



7 The formulas oflvan der Sluis 1975 p. 251) contain in his notation R(xq) / '<t„, which is the present V. 



8 The notation col (^4) is the column space of A. 
'Because A has full column rank, Ax and x can only vary proportionally when their directions are fixed. 
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3.3. Conditioning example. This example illustrates the independent effects of k 2 , V, 
and 8 on Xx(A). It is based on the example of Golub and Van Loan ( 1996 p. 238). Let 





' 1 " 




/9cos(</>) 




' " 


A = 


a 


, b = 


fi sin(0) 


, AA = 












1 




e 



where 0<e«a,(3<l. In this example, 



/3 cos((/>) 
I sin(0) 



a; + Aa; 



The three terms in the condition number are 

1 1 



x /[acos( ( /))] 2 + [sin(<£)] s 



/3cos(<^>) 

e+Q/3 sin(</>) 
a 2 +e 2 



tan(0) = 



These values can be independently manipulated by choosing a, f3, and cj). The tight upper 
bound for the condition number is 



Xx(A) < 



1 



(3yJ[acos{<j))] 2 + [sm((f))} 



The relative change to the solution of the example 

l|Ax|| 2 _ e 



+ 0(e 2 ). 



\\xh a/3 v /[acos(</>)] 2 + [sin(0)] 2 
is close to the bound given by the condition number estimate and the relative change to A. 
4. Derivation of the condition numbers. 

4.1. Notation. The formula for the Jacobian matrix J x (b) of the solution x = (A t A)~ 1 b 
with respect to b is clearp*] For derivatives with respect to the entries of A, it is necessary to 

10 The notation of section [2!2| would introducing a name, /a, for the function by which x varies with b when A is 
held fixed, x = /2 (b), so that the notation for the Jacobian matrix is then Jf 2 (b). This pedantry will be discarded 
here to write J x (b) with A held fixed; and similarly for J x (A) with b held fixed. 
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use the "vec" construction to order the matrix entries into a column vector; vec(i?) is the 
column of entries B t j with i, j in co-lexicographic orderp] The approximation is then 

Ax = J x [vec(A)] vec( A A) + J x (6) Ab + higher order terms (4. 1) 

and upon taking norms 

\\Ax\\ < || J x [vec(A)]\\ \\AA\\ + \\J x (b)\\ ||A6|| + o(max{||AA||, ||A6||}), (4.2) 

Xx{A) X x{b) 

where it is understood the norms on the two Jacobian matrices are induced from the following 
norms for AA, Ab, and Ax. 



4.2. Choice of Norms. Equation (4.2 1 applies for any choice of norms. In theoretical 
numerical analysis especially for least-squares problems the spectral norm is preferred. For 
2-norms the matrix condition number of A 1 A is the square of the matrix condition number of 
A. The norms used in this paper are thus, 



| vec (A A) | 



\AA\ 



A 



\Ab\\ 



\Ab\\ 



\Ax\ 



I Axil 



x 



(4.3) 



A 11 " B X 

where A, B, X are constant scale factors. These formulas define norms for m X n matrices, 
for m vectors, and for n vectors. The scaling makes the size of the changes relative to the 
problem of interest. The scaling used in equations (|3.1f|3.3[> is 



A=\\A\\ 



B = \\b\\ 



X = \\x\ 



(4.4) 



4.3. Condition number of x with respect to b. From x = (A 1 A) 1 A t b follows J x (b) 
(A t A)~ 1 A t and then for the scaling of equation (4.4 1 



\Ub)\\ 



\\Jx(b) A6|U 
max ,, . , , = max 

Ab ||A6|| B Ab 



B 



\Ax\\ 



\Ab\\ 



X f7„ 



(4.5) 



\Ax\\ 2 1 1 ^ 1 1 2 cr n 



= sec(0) V . 



4.4. Condition number of x with respect to A. The Jacobian matrix J a [vec(A)] is 

most easily calculated from the total differential of the identity F — A*(b — Ax) = with 
respect to A and x, which is Jp[vec(A)] vec(cL4) + Jf(x) dx = 0. Hence 



dx = -[J F {x)] 1 J F [vec(A)] vec(dA) 



(4.6) 



J x [vec(A)] 



where Jf{x) 



-A t A and where 



Jp\yec(A)] 



[ x x A l ■ ■ ■ x n A l ] , 



(4.7) 



in which r = b — Ax is the least-squares residual, and X{ is the i-th entry of x. 



1 1 The alternative to placing the entries of matrices into column vectors is to use more general linear spaces and 
the Frechet derivative. That approach seems unnecessarily abstract because the spaces have finite dimension. 
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4.5. Transpose formula for condition numbers. The desired condition number is the 



norm induced from the norms in equation (4.3 1 



" 1 V ;J " ||vec(AA)|U 

(4.8) 

_ A ||J a [vec(A)]vec(AA)|| 2 

— — nicix r. 

X aa \\AA\\ 3 

The numerator and denominator are vector and matrix 2-norms, respectively. If A is an m x n 
matrix, then the maximization in equation ( |4.8| l has many degrees of freedom. An identity 
for operator norms can be applied to avoid this large optimization problem. 

Suppose M. AI and R N have the norms || • \\m and |j • ||jy, respectively. If a problem with 
data d € R N has a solution function s = f(d) € R M , then the condition number is the 
induced norm of the M x N Jacobian matrix, 

\\J f (d)Ad\\ M 

W J fWW = *T? \\Ad\\ N ■ (49) 

This optimization problem has N degrees of freedom. An alternate expression is the norm 
for the transposed operator represented by the transposed matrixf^] 

\\Jf(d)\\ = \\[Jf(d)]T= r r lllJ y i " ' (4 ' 10) 

As ||As|| M 

The norm is induced from the dual norms || • \\* M and || • ||^ which must be determined. This 



optimization problem has M degrees of freedom. Equation ( 4. 10 1 might be easier to evaluate, 
especially when the problem has many more data values than solution variables, N M, as 
is often the case. 

Applying the formula ( |4.10| i for the norm of the transpose matrix to the equation ( |4.8| l 
results in the simpler optimization problem, 

X Ax 1 1 Z\cc 1 1 5 

The norm for the transposed Jacobian matrix is induced from the duals of the 2-norms for 
matrices and vectors. The vector 2-norm in the denominator is its own dual. So as not to 
interrupt the present discussion, some facts needed to evaluate the numerator are proved in 
section [6] the dual of the matrix 2-norm is the nuclear norm (section [63) , and a formula for 



the nuclear norm is given (section 6.4 1 



4.6. Condition number of x with respect to A, continued. The application of equation 



(4. 1 1 1 requires evaluating the matrix-vector product in the numerator. Continuing the deriva- 
tion of section 4.4 from equation (4.7 1, the vector-matrix product v t Jp[vsc(A)] for some v 
can be evaluated by straightforward multiplication, 

v t J F [vsc(A)] = [ vir* ••• v n r t ]-[xiv t A t ■■■ x n v t A t ]. 

12 Equation |4.10} is stated by |Bourbaki|jl987| chp . IV, p 7, eqn. 4), |Dunford and Schwartz]|l958| p. 478, lem. 
2), |Rudin|(l973| p. 93, thm. 4.10, eqn. 2), and |YosidaHl974[ p. 195, thm. 2', eqn. 3). The name of the transposed 
operator varies. See Bourbaki 1 1987 chp. IV, p. 6, top) for "transpose," Dunford and Schwartz (loc. cit.) and Rudin 
(loc. cit.) for "adjoint," and Yosida ( 1974 p. 194, def. 1) for "conjugate" or "dual." Some parts of mathematics use 
"adjoint" in the restricted context of Hilbert spaces, for example in linear algebra see Lancaster and Tismenetsky 
j!985| pp. 168-174, sec. 5.1). That concept is actually a "slightly different notion" (Dunford and Schw artz||1958| p 
479) from the Banach space transpose used here. 
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This row vector, when transposed, is expressed more simply using vec notation: the first part 
is r scaled by each entry of v, vec(ru*), the second part is Av scaled by each entry of x, 
vec(Avx t ). Altogether 

J F [vec(.A)]'^ — vec (7V — Avx 1 ) . 



Substituting v = {-[Jf^)]- 1 }* Ax = (A t A)~ 1 Ax for some Ax gives, by equation ( |4.6[ ), 

J x [vec(A)]*Ax = vec {r {(A* Ay 1 Ax]* - A [(A* A)' 1 Ax] x 1 } , 
or equivalently, 

mat{J x [vec(A)]*Aa;} = u^* + u 2 v\ (4.12) 
where "mat" is the inverse of "vec," and 

m = r = b- Ax, vi = (A* A)' 1 Ax, 

(4 13) 

u 2 = -A(A t A)- 1 Ax, v 2 =x. 



The matrix on the right side of equation (4.12 1 has rank 2. Moreover, the two rank 1 pieces 



are mutually orthogonal because the least-squares residual r is orthogonal to the coefficient 
matrix A. With these replacements equation ( |4.1 1[ ) becomes 

A 

J,[vec(A)] = - max \\ Ul vl + u 2 vl\\* 2 . (4.14) 

A. jAxj[2 — 1 

Lemma |6~2|shows that the dual of the spectral matrix norm is the matrix norm that sums 



the singular values of the matrix, which is called the nuclear norm. Lemma 6.3 then evaluates 



this norm for rank 2 matrices to find that the objective function of equation (4.14 1 is 



IHIi + IMI! IMII + 2 ||ui|| 2 IHb IW2 IM2 cos(6> n - 9 V ) , (4.15) 

where 6 U is the angle between U\ and u 2 , and 6 V is the angle between V\ and v%, and both 
angles should be taken from to ir. Since U\ is orthogonal to u 2 therefore 8 U — ir/2 and then 
\9 U — 6 V I < 7r/2 so cos(8 u — 6 V ) is not negative. This means the maximum lies between the 
lower and upper limits 

VWllilKIH + IMHIM! and IKIUIKHa + lluablWb. (4-16) 

With || Ax||2 restricted to 1, the lower bound and also the upper bound attain their maxima 
when Ax is a right singular unit vector for the smallest singular value of A, 

||t*i||a|h||2 = 7 J ! % and \\u 2 \\ 2 \\v 2 \\ 2 = & . (4.17) 

V^minJ ^"min 

Some value of \\u l v\ + u 2 v 2 \\ 2 lies between the limits when Ax is a right singular unit 
vector for the smallest singular value of A. Because these are the largest possible limits, 
the maximum value must lie between them as well. These limits must be multiplied by the 



coefficient A/X in equation (4.14i to obtain bounds for the norm of the Jacobian matrix. 
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4.7. Summary of condition numbers. 

Theorem 4.1 (Spectral condition numbers). For the full column rank linear 
least-squares problem with solution x — (A t A)~ 1 A t b, and for the scaled norms of equation 
\4.3\ with scale factors A, B, and X, 

B A 

Xx(b) = - Xx(A) = — max a x + o 2 , 



where o\ and a 2 are the singular values of the rank 2 matrix u l v\ + u 2 v 2 for 

u\ = r = b- Ax, vi = (A t A)~ 1 Ax, 

u 2 = -A(A t A)~ 1 Ax, v 2 =x. 

The value of Xx{A) lies between the lower limit of Malyshev and the upper limit ofBjdrck, 




+ Nil < Xx{A) < 



A 



X(Ty, 



+ NI2 



The upper bound exceeds \x (A) by at most a factor v2. The formula for \x (b) and the limits 



for Xx(A) simplify to equations ( 3.1\ 3.2\ for the scale factors in equation ( |4.4| ). 

Proof. Section 4.3 derives Xx(b), and sections 4.4-4.7 derive Xx(A) and the bounds. □ 



5. Discussion. 

5.1. Example of strict limits. The condition number Xx(A) in theorem |4.1| can lie 

strictly between the limits of Bjorck and Malyshev. For the example of section [3.3| the rank 
2 matrix in the theorem is 



— (3 cos(<fi) Axi —f3s'm((j))Axi/a 
-f3co$(4>)Ax2/a —(3sin(4>)Ax 2 /a 2 
Axi Ai 2 /a 2 



For the specific values a — 1/10, j3 = 1, <f> = tt/10, the sum of the singular values of this 
matrix can be numerically maximized over 1 1 Ax \ \ 2 — 1 to evaluate the condition number with 
the following results. 



K 2 (Vtan(0) + 1) 

Xx(A) 

k 2 ([Vtan(0)] 2 - 



= 40.928. 
= 35.193. 
1) 1/2 = 32.505. 



upper limit of Bjorck 
condition number 
lower limit of Malyshev 



These calculations were done with Mathematica ( Wolf ram] |2003| l . 

5.2. Exact formulas for some condition numbers. Table 15.11 lists several condition 
numbers or approximations to condition numbers for least-squares solutions. The three exact 
values measure changes to A by the Frobenius norm, while the two approximate values are 
for the spectral norm. The difference can be attributed to the ease or difficulty of solving 
the maximization problem in equation ( |4.14| >. The dual spectral norm of the rank 2 matrix 
involves a trigonometric function, cos(7r/2 — 9 V ) in equation (4. 15 I, whose value only can be 
estimated. If a Frobenius norm were used instead, then lemma |63| shows the dual norm of the 
rank 2 matrix involves an expression, cos(7r/2) cos(0„), whose value is zero, which greatly 
simplifies the maximization problem. 



10 



JOSEPH F. GRCAR 



Table 5.1 

Cases for which condition numbers have been determined for the full column rank least-squares problem, 
min^ ||b — j4x|| 2- All the formulas are for Xx(A) except Grattan's formula is for Xx[A, b). Notation: r is the 
residual, tr m i n is the smallest singular value of A. In column 5, "approx" means the value in column 4 approximates 
the condition number, "exact" means it is the condition number for the chosen norms. 



data 



solution 



formula 



4.1 



Bjorck and theorem 

(1996, p. 31, eqn. 1.4.28) 



Geurts 

(1982, p. 93, eqn. 4.3) 



Gratton 

(1996, p. 525, eqn. 2.1) 



Malyshev 

(2003, p. 1187, eqn. 1.8) 



Malyshev and theorem 

(2003, p. 1 189, eqn. 2.4 and lme^S) 



|AA|| : 




Axil 



N|2 

Adla 



Nla 
Ax]| 2 



Aa|| 2 



0"min \ CTmin ||^||2 

PIIf 



+ 1 



^min V ^min 11*111 



+ 1 



+ 



> J + /3 2 



\A\[ 



+ 1 



5.3. Overestimates of condition numbers. Many error bounds in the literature com- 
bine Xx(A) + Xx(b) in the manner of equation (2.6 1, 

„ „ 2 < Xx(A, b)e + o(e) attainable 

< [xx(A) + Xx(b)]c + o(e) overestimate by at most x 2 (5.1) 

< [re 2 (vtan(0) + 1) + vscc(0)]e + o(e) further at most x y/2 (5.2) 
Ah\\r\\ 2 , ||A|| 2 , ||b|| a 



x 2 



\ x h 



CT mirl 1 1 ^ 1 1 2 



o(e) 



(5.3) 



where e = max{|| AA||2/||A||2, ||A6||2/||6||2}. Bounds (5.1 5.2 1 are la rger than the atta in 



able bound by at most factors 2 and 2\/2, respectively, by equation (2.7 i and theorem 4.1 
Some bounds are yet larger. High am| ( |2002 p. 382, eqn. 20.1) reports 



|Ax|| 2 

1RU" 



< re 2 e 2 + (re 2 + 1 



0(e 2 ) 



\\Ah\\rh 

a Ln\\ X h 



Ah\\x\\ 2 _ 
Ah , IH| 2 ||a:||2 + ||r||a 



\ x \\ 2 



0(e 2 ) 



This bound is an overestimate in comparison to equation ( 5.3 1. 

An egregious overestimate occurs in an error bound that appears to have originated in the 
1983 edition of the popula r textbook of[GoTub and Van Loan| ( [T996| p. 242, eqn. 5.3.8). The 
overestimate is restated by | Anderson et al. ( 1992| p. 50) in the LAPACK documentation, and 
by |Demmel| ( [T996l p. 117), 



|As|| ; 



< [2 sec(0) k 2 + tan(0) re 2 ,] e + 0(e 2 



(5.4) 
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In comparison with equation (5.2 1 this bound replaces V by k 2 and replaces 1 by sec(0). An 
overestimate by a factor of k 2 occurs for the example of section 3.3 with a <C 1, j3 = 1, and 



In this case the ratio of equation (5.4 1 to equation (5.2 1 is 



i + 2 av /r 

a 2 P 



2 sec(O) k 2 + tan(0) k 2 
K 2 (Vtan(0) + l)+Vscc(0) ~ 1 + [3 + ay/T+J* 



1 

2a 



ad 



6. Norms of operators on normed linear spaces of finite dimension. 

6.1. Introduction. This section describes the dual norms in the formulas of sections 
|4.5| and |4~6l The actual mathematical concept is a norm for the dual space. However, linear 
algebra "identifies" a space with its dual, so the concept becomes a "dual norm" for the same 
space. This point of view is appropriate for Hilbert spaces, but it omits an important level 
of abstraction. As a result, the linear algebra literature lacks a complete development of 
finite dimensional normed linear (Banach) spaces. Rather than make functional analysis a 
prerequisite for this paper, here the identification approach is generalized to give dual norms 
for spaces other than column vectors (which is needed for data in matrix form), but only as 



far as the dual norm itself in section 6.2 Section 6.3 gives the formula for the dual of the 



spectral matrix norm. Section 6.4 evaluates the norm for matrices of rank 2. 

Banach spaces are needed in this paper because the norms used in numerical analysis 
are not necessarily those of a Hilbert space. The space of m x n matrices viewed as column 
vectors has been given the spectral matrix norm in equation ( |4.3[ ). If the norm were to make 
the space a Hilbert space, then the norm would be given by an inner product. There would be 
an van x mn symmetric matrix, S, so that for every m x n matrix B, 



\B\\ 



vec(B)Y S vec( J B) , 



which is impossible. 



6.2. Duals of normed spaces. If X is a finite dimensional vector space over R, then the 
dual space X* consists of all linear transformations / : X — >• M, called functionals. If X has 
a norm, then X* has the usual operator norm given by 



sup f(x). 

11x11=1 



(6.1) 



One notation is used for both norms because whether a norm is for X or X* can be decided 
by what is inside. 

For a finite dimensional X with a basis e\, e-x, . . . , e n , the dual space has a basis g\, 
g2, f n defined by fi(ej) — Sij where 5i.j is Rronecker's delta function. In linear 
algebra for finite dimensional spaces, it is customary to represent the arithmetic of X* in 
terms of X under the transformation T : X —> X* defined on the bases by T(ei) = /j. 
This transformation is not unique because it depends on the choices of bases. Usually X 
has a favored or "canonical" basis whose T is said to "identify" X* with X. Under this 
identification the norm for the dual space then is regarded as a norm for the original space. 

Definition 6.1 (Dual norm). Let X = W m have norm || • |j and let T identify X 
with the dual space X*. The dual norm for X is 



\T{v)\\ 
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where the right side is the norm in equation \6.1\ for the dual space. 

The notation || • || * avoids confusing the two norms for X. There seems to be no standard 
notation for the dual norm; others are || ■ ||d, || • \\ D , and || • ||a which are used respectively by 



Higham(2002 



and Ti smenetsky|( |T985l p. 381, eqn. 1). 



p. 107, eqn. 6.2), Horn and Johnson ( 1985 p. 275, def. 5.4.12), and Lancaster 



6.3. Dual of the spectral matrix norm. The space R mx ™ of real mxn matrices has 
a canonical basis consisting of the matrices E^ 1 '^ whose entries are zero except the i, j entry 
which is 1. This basis identifies a matrix A with the functional whose value at a matrix B is 

Lemma 6.2 (Dual of the spectral matrix norm). The dual norm of the spec- 
tral matrix norm with respect to the aforementioned canonical basis for ]R mxn is given by 
\\A\\2 — ||ct(j4)||i, where o(A) € jjmin{m,n} - s j n£ vec t or f A' s singular values including 
multiplicities. That is, ||A|| 2 is the sum of the singular values of A with multiplicities, which 
is called the nuclear norm or the trace norm. 

{2003).) Let A = UYiV* be a "full" singular value decompo- 



Proof. (Supplied by 



Kahan 



sition of A, where both U and V are orthogonal matrices, and where £ is an m x n "diagonal" 
matrix whose diagonal entries are those of a (A). The trace of a square matrix, M, is invariant 
under conjugation, V~ X MV, so 



\A\\ 



sup \x{A t B)= sup ti(V'£ t U t B) = sup tr (£' V t B V) . 



'I S II 2 = 1 



i|S|| 2 = l 



|B||2 = 1 



Since ||[/*.BV||2 = 1 1 -B 1 1 2 = 1, the entries of U t BV are at most 1 in magnitude, and therefore 
I tr (£' V) I < tr (£*). This upper bound is attained for B = UDV 1 where D is the m x n 
"identity" matrix. □ 

An alternate proof is offered by the work of von |von Neumann] ( |1937| l. He studied a 
special class of norms for H. mXn . A symmetric gauge function of order p is a norm for R p 
that is unchanged by every permutation and sign change of the entries of the vectors. Such 
a function applied to the singular values of a matrix always defines a norm on M mx ™. For 
example, ||A||2 = ||o"(^4)||oo where as in lemma 6.2 a(A) is the length min{m,n} column 



vector of singular values for A. The dual of this norm is given by the dual norm for the 
singular values vector, see |Stewart and Siin| ( |1990| p. 78, lem. 3.5). 

Proof. (In the manner of John von Neumann.) By the aforementioned lemma to von 
Neumann's gauge theorem, ||A||2 = ||f(^) llSo = IkC^lli- ^ 

6.4. Rank 2 Matrices. This section finds singular values of rank 2 matrices to establish 



some norms of the matrices that simplify the condition numbers in equation (4. 14 1. 

Lemma 6.3 (Frobenius and nuclear norms of rank 2 matrices.). Ifm,U2 e 
M. m and Vi,V2 € K™, then (Frobenius norm) 



\uj_vl +u 2 v 2 \\ 



iilll IHI! + \\u 2 \\l IMII + 2 |K|| 2 |K|| 2 ||u 2 || 2 ||v 2 || 2 cos(e u )cos(e„) , 



and (nuclear norm, or trace norm) 
IK«l +U2V2WI = 

ll^illi + Mi iwii + 2 IM| 2 IM2 \\u2W2 \\v2W2 cos(e u 



where 8 U is the angle between Ui and m 2 , and 9 V is the angle between V\ and u 2 . Both angles 
should be taken from to it. 
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Proof. If any of the vectors vanish, then the formulas are clearly true, so it may be 
assumed that the vectors are nonzero. The strategy of the proof is to represent the rank 2 
matrix as a 2 x 2 matrix whose singular values can be calculated. Since singular values are 
wanted, it is necessary that the bases for the 2 x 2 representation be orthonormal. 

To that end, let w\ and W2 be orthogonal unit vectors with u\ = a\W\ and — a.2W\ + 
(iw2- The coefficients' signs are indeterminate, so without loss of generality assume ot\ > 
and (3 > 0, in which case 



«1 2 



OL2 



"1"2 
U1II2 



P 



U2 



\U1W2 



Similarly, let x\ and x 2 be mutually orthogonal unit vectors with v\ 
72^1 + 5x2- Again without loss of generality 71 > and 5 > so that 



71X1 and V2 = 



7i = II ^1 II 2 



72 



Notice that 



/? 2 = ll 



U2\\l 



u l v 2 

\\vih 



\Wih 



V2 



u l u 2 

\vih 



«1 2 



Ml 



l u 2 



\\vi\ 



Let G — u 1 v\+ u^v%. A straightforward calculation shows that, with respect to the orthonor- 
mal basis consisting of X\ and x 2 , the matrix G t G is represented by the matrix 



M = 



1 72 2 + (ui 7i + « 2 72) 2 P 2 S-f 2 + S a 2 (ax 71 + a 2 72) 
P 2 5^2 + Sa 2 (ai7i +(2272) P 2 S 2 + S 2 a 2 2 

The desired norms are now given in terms of the eigenvalues of M, \±, 



l|G|| F = v^i 



x/tr(M) and j|G|| 



The expression for ||G||2 requires further analysis. For any 2x2 matrix M, 



^) -det(M,. 



In the present case these eigenvalues are real because the M of interest is symmetric, and 
det(M) > because it is also positive semidefinite. Altogether [tr(M)] 2 > 4det(M) > 0, 
so tr(M) > 2det(M) > 0. These bounds prove the following quantities are real, and it can 
be verified they are the square roots of the eigenvalues of M, 



+ 



tr(M) /det(M) /tr(M) 



± 



det(M) 



thus 



In summary, the desired quantities ||G||2 and ||G|| F have been expressed in terms of 
det(M) and tr(M) which the expression for M expands into formulas of ai, P, -fi, and 8. 



\\G\\*2 = v^7+ V^= \/ tr ( M ) + 2 V / det(M). 
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These in turn expand to expressions of Ui and t>j. It is remarkable that the ultimate expressions 
in terms of Uj and Vi are straightforward, 

tr(M) = Muxlla + 1 1 ^2 1 1 1 IHI2 + 2 {u\u 2 ){v\v 2 ) 

= ||fx||l + IMI2 ll«2||i + 2 IKIh II 2 IK|| 2 1|-^2 II 2 cos(6» u )cos(6» 1 ,) 

det(M) - (|| Ul ||= \\u 2 \\l - (u\u 2 f) \\v 2 \\l - {v\v 2 f) 

= (|| Ul ||2|| U 2||^(sin(^)) 2 )(||T; 1 ||2|| V2 ||2( s in(^)) 2 ) , 

where 9 U is the angle between u\ and u 2 , and similarly for 8 V . The formula for ||G||f is 
established. The formula for ||G||2 simplifies, using the difference formula for cosine, to the 
one in the statement of the lemma. Since the positive root of ^det(M) is wanted, the angles 
should be chosen from to tt so the squares of the sines can be removed without inserting a 
change of sign. These calculations have been verified with Mathematica (Wolfram, 2003). □ 
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